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Abstract 


The  purpose  of  this  study  was  to  develop  further  sophistication  in  the  understanding 
and  exposition  of  the  impact  of  the  Health  Technology  Assessment  (HTA)  Unit's 
products.  Data  were  collected  through  a series  of  open-ended  questions  during  face-to- 
face  interviews,  then  analyzed  using  a qualitative  approach  in  which  common  themes 
were  extracted  to  develop  conclusions  about  the  impact  of  HTA  products  on  decision- 
making. It  was  found  that  evidence  presented  in  HTA  products  informs  decisions  of 
the  requesting  organization.  Factors  that  have  a positive  influence  on  impact  include 
the  quality  of  the  report  and  the  reputation  and  credibility  of  the  HTA  Unit.  Time  of 
availability  of  the  product  can  affect  the  impact  of  the  product.  In  order  for  products  to 
have  a positive  impact,  they  must  be  available  at  the  time  the  decision  is  being  made. 
Products  that  present  inconclusive  evidence  have  less  impact  on  decision-making  and 
the  presence  of  external  barriers  (e.g.,  advocacy  action  of  lobby  groups  and  the 
momentum  of  existing  programs)  may  further  limit  their  impact. 
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Purpose 


HTA  evaluates  the  properties  and  effects  of  health  care  technology1  and  provides 
information  to  support  all  health  care  decisions  at  local,  regional,  national,  and 
international  levels.  As  such,  the  HTA  Unit  at  the  Alberta  Heritage  Foundation  for 
Medical  Research  (AHFMR)  endeavours  to  provide  information  on  the  safety,  efficacy, 
effectiveness,  and  economic  impact  of  health  technologies  to  support  health  care 
decisions  and  policy  making.  The  Unit's  Report  of  Activities  for  1999-2000 2 attempted  to 
identify  the  impact  of  the  unit's  products  produced  during  the  course  of  that  fiscal  year, 
on  the  practice,  decisions  and/or  policy  of  the  clients  who  had  requested  the  work. 
These  impacts  were  descriptive  and  brief. 

To  that  same  end,  the  Unit  engaged  the  services  of  an  external  consultant,  TurnKey 
Management  Consulting,  to  study  the  impact  of  the  Unit's  2000-2001  products  (Table  1 
lists  the  products  considered  in  this  study.)  The  results  of  the  project  will  guide  the 
continuous  improvement  efforts  of  the  unit  and  will  be  included  in  the  2000-2001  Report 
of  Activities.  It  is  also  expected  that  the  work  will  serve  as  a template  to  be  repeated  and 
refined  in  subsequent  years. 

Table  1:  HTA  products  assessed  in  this  study 


Title  of  Product3 

Type  of  Product4 

Date 

Mammography  screening:  mortality  rate  reduction  and  screening 
interval 

Report 

June  2000 

Visudynetm  therapy  for  the  treatment  of  age-related  macular 
degeneration 

Technote 

July  2000 

Chemoembolization  treatment  for  colorectal  metastases  to  the 
liver 

Technote 

October  2000 

Conductive  education  for  children  with  cerebral  palsy 

Report 

November  2000 

Laproscopic  adjustable  gastric  banding  for  clinically  severe 
(morbid)  obesity 

Brief 

December  2000 

Extremity  pumps  for  treatment  of  primary  peripheral  edema 

Technote 

January  2001 

Intensive  intervention  programs  for  children  with  autism 

Brief 

February  2001 

Patient  diabetes  education  in  the  management  of  adult  type  2 
diabetes 

Report 

February  2001 

Off-pump  coronary  artery  bypass  surgery 

Technote 

March  2001 

Vagus  nerve  stimulation  for  refractory  epilepsy 

Report 

March  2001 

1 Health  technologies  include  drugs,  devices,  medical  and  surgical  procedures,  and  the  administrative  and  support 
systems  in  which  health  care  is  delivered. 

2 Juzwishin,  D.  (2000).  Health  Technology  Assessment  Unit  Report  of  Activities  for  1999-2000.  HTA  Annual  Reports: 
1999/00:  pp.  4,  8,  10,  11,  12-14.  Available  at  http://www.ahfmr.ab.ca/  publications.htmi. 

3 The  individual  requesters  of  HTA  products  considered  by  this  study  are  purposely  not  identified  in  order  to 
safeguard  the  anonymity  of  the  study’s  interviewees.  However,  organizations  requesting  at  least  one  of  the  products 
listed  in  Table  1 may  be  identified,  and  these  included  the  Alberta  Medical  Association,  Alberta  Learning,  Alberta 
Health  and  Wellness,  Capital  Health  Authority,  Alberta  Children’s  Services,  and  Calgary  Health  Region. 

4 Health  Technology  Assessment  Report.  Detailed  appraisals  of  health  technologies;  providing  a synthesis  of  data 
from  the  literature,  or  reporting  on  empirical  studies.  Subject  to  external  review.  Health  Technology  Assessment 
Brief.  Limited  assessments  providing  concise  advice  on  technologies.  Subject  to  external  review.  Technote:  Brief 
responses  to  requests  for  rapid  advice,  with  limited  analysis.  These  are  not  subject  to  external  review. 
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Methodology 

This  section  describes  how  information  was  gathered  and  analyzed.  Data  were 
collected  through  a series  of  open-ended  questions  during  face-to-face  interviews.  A 
qualitative  analysis  was  used  to  extrapolate  common  themes  and  develop  conclusions 
about  the  impact  of  HTA  products  on  decision-making.  This  approach  was  chosen  for 
the  following  reasons: 

• There  were  insufficient  numbers  of  products  to  conduct  a meaningful 
quantitative  assessment  of  impact. 

• Since  this  was  the  first  attempt  at  impact  assessment,  the  potential  range  of 
responses  was  unknown,  making  it  difficult  to  develop  multiple-choice  type  of 
questions  with  defined  answers.  Also,  providing  defined  responses  to  questions 
could  potentially  bias  responses. 

• Face-to-face  interviews  ensured  that  respondents  participated  in  the  study,  as 
opposed  to  a mail-in  survey  that  could  easily  be  ignored.  The  interview  also 
permitted  more  in-depth  questioning  and  the  opportunity  to  clarify  responses. 

• The  use  of  an  external  consultant  likely  encouraged  interviewees  to  give  more 
candid  responses. 

Interview  Guide 

The  ultimate  goal  of  this  study  was  to  confirm  and  expand  the  HTA  Unit's 
understanding  of  its  products'  impact.  Hailey  et  al.5  identified  a number  of  challenges 
to  the  assessment  of  HTA  impact,  including: 

• the  multiplicity  of  influences  on  the  policy-making  process  other  than  HTA;  and 

• the  difficulty  of  measuring  the  longer-term  impacts  of  HTA. 

Recognizing  these  challenges,  this  study  of  HTA  impact  included  three  types  of 
evaluation:  context,  implementation,  and  outcome6.  Context  evaluation  identified 
environmental  enablers  and  barriers  that  influenced  the  impact  of  HTA  products; 
implementation  evaluation  identified  critical  processes  and  activities  in  producing  HTA 
products  that  influenced  impact;  and  outcome  evaluation  identified  the  extent  to  which 
the  HTA  products  influenced  health  care  policy  and  decision  makers.  Given  that  some 
of  the  products  considered  in  this  study  were  less  than  a year  old  at  the  time  of  the 
study,  the  influence  of  each  on  the  health  care  system  was  not  expected  to  be  evident.  It 
was  therefore  important  to  identify  "intermediate"  steps  in  achieving  the  final  outcome. 
As  such,  two  "outcome"  questions  were  included  in  the  questionnaire:  one  explored 


5 Hailey,  D.M.,  D.E.  Cowley,  W,  Dankiw  (1990).  The  impact  of  health  technology  assessment.  Community  Health 
Studies:  15  (3)  223-234. 

6 Peterson,  A.C.  (1998).  W.K  Kellogg  Foundation  Evaluation  Handbook,  pp.  20-46.  Available  at 
http://www.wkkf.org/pubs/Pub770.pdf. 
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how  the  product  had  been  used  in  the  short-term  and  the  second  explored  the  future 
potential  for  the  product.  Appendix  1 lists  the  questions  that  interviewees  were  asked. 

Interviews 

The  HTA  Unit  identified  prospective  interviewees  as  either  the  original  requester  of  a 
product  or  individuals  who  approached  the  HTA  Unit  during  the  development  of  a 
product  with  a similar  request  (i.e.,  they  would  have  requested  the  product  had  it  not 
already  been  requested).  For  the  ten  products,  18  different  individuals  were  identified 
and  three  of  these  18  were  associated  with  two  products.  Some  products  had  only  one 
contact  and  two  of  the  products  had  four  contacts. 

The  HTA  Unit  extended  email  invitations  to  the  selected  contacts  to  participate  in  the 
study.  Appendix  2 contains  an  example  of  the  message  that  was  sent  to  prospective 
interviewees.  One  individual  declined  the  interview,  deferring  to  another  contact  and 
one  individual  requested  that  a colleague,  who  was  involved  in  the  implementation  of 
the  conclusions  of  the  HTA  product,  attend  the  interview  instead.  All  other  individuals 
accepted  the  invitation  to  the  interview,  although  many  had  difficulty  meeting  on  short 
notice.  A time  was  scheduled  for  the  interview  and  interviewees  received  a copy  of  the 
interview  guide  (see  Appendix  1)  prior  to  the  interview. 

Of  the  17  interviews  conducted,  11  were  conducted  in  person  and  six  were  conducted 
via  telephone.  Two  interviewers  attended  all  interviews,  except  for  three  interviews 
that  were  attended  by  only  one  interviewer.  One  interviewer  asked  questions  while  the 
other  manually  took  note  of  responses.  The  interviews  took  approximately  30-40 
minutes  to  complete.  Where  interviewees  served  as  a contact  for  two  products,  the 
interviews  were  conducted  consecutively.  Interviewees  were  given  the  opportunity  of 
commenting  on  a draft  version  of  this  report  prior  to  its  release  to  ensure  that 
interviewee  comments  were  reported  and  interpreted  accurately. 

Analysis 

Interview  responses  were  grouped  into  the  topics  listed  in  Table  2.  For  each  product, 
interview  responses  were  then  assigned  to  one  of  these  topics.  This  analysis  generated 
a table  of  comments  made  about  a specific  product.  Responses  to  the  questions 
indicated  in  the  right  hand  column  in  Table  2 informed  the  corresponding  topic.  The 
responses  were  then  themed  and  summarized  to  present  a list  of  comments  and  the 
number  of  products  to  which  the  comment  applied  (see  Table  3.) 
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Table  2:  Topics  used  to  guide  data  analysis 


Topic 

Information  Definition 

Questions 

Reason  for  Request 

Why  the  requesting  organization  requested  the  product  and 
the  information  they  needed  in  order  to  make  a decision 

1,2,3 

External 

Barriers/Enablers 

"Environmental"  factors  that  may  have  influenced  the  ability 
of  the  requesting  organization  to  use  the  conclusions  of  the 
HTA  product 

5,  6 

Internal 

Barriers/Enablers 

Factors  and  structures  within  the  requesting  organization  that 
may  have  influenced  the  ability  of  the  requesting  organization 
to  use  the  conclusions  of  the  HTA  product 

5,  6 

Process 

Barriers/Enablers 

Specific  aspects  of  the  process  for  requesting  and  developing 
the  product  that  may  have  influenced  the  ability  of  the 
requesting  organization  to  use  the  conclusions  of  the  product 

7 

Immediate  Outcome 

How  the  requesting  organization  used  the  product  and  how 
the  product  impacted  decisions. 

4,8 

Future  Potential 

A subjective  opinion  of  how  the  requesting  organization  might 
use  the  product  in  the  future  and  how  the  product  may  inform 
future  decisions. 

9 
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Findings 

This  section  presents  the  data  collected  in  the  interviews  and  discusses  its  relevance. 

Interview  Responses 

Responses  to  interview  questions  were  categorized  into  themes  and  enumerated.  Table 
3 lists  the  responses  for  the  themes  and  the  number  of  times  the  response  related  to 
individual  products. 

Table  3:  Interview  comments  about  HTA  reports  grouped  according  to  topic 

(The  “number  of  products”  is  the  number  of  products  the  comment  pertains  to.) 


Number  of 
Products 

Comments 

Reasons  why  the  HTA  product  was  requested. 

5 

Funding  claims  made  to  Alberta  Health  and  Wellness  where  evidence  was  required  to  make  a 
decision  on  whether  to  fund  individual  claims  and  whether  to  publicly  fund  the  treatment. 

3 

Evidence  required  to  make  decisions  about  program  funding,  continuation  of  program,  and 
elements  of  program  delivery. 

1 

Evidence  required  to  make  decision  about  changes  to  clinical  practice  guidelines 

1 

Evidence  required  to  make  decision  about  patient  care 

External  Barriers 

3 

Lobbying  by  advocacy  group  influenced  policy,  funding  and  program  decisions 

2 

Upcoming  provincial  election  influenced  policy,  funding  and  program  decisions 

2 

Difficult  to  cut  funding  for  established  programs 

1 

Court  ruling  set  precedence  which  influenced  decision  to  offer  programs 

1 

Current  program  elements  set  precedence  to  continue  offering  a program  incorporating  those 
elements 

1 

Practices  of  other  provinces  influenced  decisions  to  establish  programs 

1 

Different  treatment  undertaken  negated  need  to  apply  technology 

1 

Program  did  not  strictly  comply  with  established  models  of  program  delivery  making  it  difficult  to 
draw  parallels  about  program  efficacy 

1 

Unanswered  guestions  about  data  ownership 

External  Enablers 

0 

None  reported 

Internal  Barriers 

2 

Individual  reguesting  product  acting  on  own  initiative  and  did  not  communicate  with  stakeholders 

1 

Difficulty  responding  proactively  to  external  barriers 

1 

Internal  politics 

1 

Distracted  by  other  internal  events 

Internal  Enablers 

7 

Internal  standing  or  ad  hoc  committees  in  place  to  make  reguired  decisions. 

6 

Established  communication  channels 

1 

Policy  that  funding  decisions  must  be  evidence-based 
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Table  3:  Interview  comments  about  HTA  reports  grouped  according  to  topic  (cont’d) 

(The  “number  of  products”  is  the  number  of  products  the  comment  pertains  to.) 


Number  of 
Products 

Comments 

Process  Barriers 

5 

Timing 

1 

Resistance  to  inclusion  of  context  information  in  product 

1 

Product  not  posted  on  HTA  web  site 

1 

Level  of  product  not  appropriate  for  audience 

Process  Enablers 

7 

High  quality,  thorough  product 

5 

Appropriate  level  for  audience 

5 

Reputation  and  credibility  of  HTA  Unit 

Process  Enablers 

4 

Timeliness;  i.e.,  product  available  when  decision  required 

3 

Unbiased,  objective  product 

3 

Ongoing  communication  with  HTA  Unit 

2 

Assistance  provided  by  HTA  Unit  in  communicating  with  stakeholders 

2 

Format  of  the  product 

2 

Excellent  reference  list 

1 

Assistance  provided  by  HTA  Unit  in  formulating  question 

1 

Review  process 

1 

HTA  Unit  leveraged  work  of  other  HTA  products  prepared  by  other  jurisdictions 

1 

Established  relationship  between  reguesting  organization  and  HTA  Unit 

1 

Inclusion  of  context  data 

Suggestions  for  Enhancements/lmprovements  to  Service  Provided  by  HTA  Unit 

2 

HTA  Unit  should  consider  playing  a more  active  role  in  disseminating  their  products 

2 

HTA  Unit  should  consider  updating  products  on  high  priority  topics  with  the  most  current  evidence 
on  a regular  basis 

2 

; HTA  Unit  should  consider  offering  a service  that  provides  short  turn  around  (3-7days)  short  notes 
on  high  priority  topics 

1 

HTA  Unit  should  consider  offering  a "menu"  of  research  areas  that  could  be  included  in  the 
i product  e.g.,  information  about  other  jurisdictions,  law  and  policy,  etc. 

1 

HTA  Unit  should  consider  providing  information  about  public  opinion  in  their  products 

Immediate  Outcomes 

8 

Informed  policy  and  resource  allocation 

6 

Raised  awareness 

3 

No  impact  on  decisions 

2 

Provided  impetus  for  future  research 

2 

Changed  patient  care  practice 

1 

Provided  evidence  for  use  in  legal  decision 

1 

Information  contained  in  the  product  informed  changes  to  program  elements 
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Table  3:  Interview  comments  about  HTA  reports  grouped  according  to  topic  (cont’d) 

(The  “number  of  products”  is  the  number  of  products  the  comment  pertains  to.) 


Number  of 
Products 

Comments 

Future  Potential 

3 

Inform  policy  and  resource  allocation  decisions  in  the  requesting  organization 

3 

None  - served  its  purpose 

2 

Inform  policy  and  resource  allocation  decisions  in  other  jurisdictions 

2 

Future  reference 

2 

Serve  as  a model  format  for  future  HTA  product  requests 

2 

Will  request  HTA  Unit  to  update  information  at  some  future  time 

1 

Demonstrates  value  of  HTA  Unit’s  work 

1 

Organization  will  attempt  to  employ  evidence-based  decision  making  in  the  future 

1 

Changes  to  practitioner  behaviour  and  patient  care 

Recurring  variables  emerged  from  this  analysis;  e.g.,  the  same  external  barriers  were 
identified  for  several  different  products.  Table  4 lists  the  common  variables  identified 
for  the  various  topics. 

Table  4.  Variable  definitions 


Topic 

Variable 

Definition 

Product 

Attributes 

Conclusiveness 

A variable  that  indicates  whether  the  evidence  in  the  HTA  product 
provided  clear  conclusions  as  to  the  efficacy  of  the  technology. 

Peer  Reviewed 

A measure  that  classifies  the  product  into  one  of  two  types: 

• Peer  Reviewed  - Health  Technology  Assessment  Report  or  Health 
Technology  Assessment  Brief 

• Not  Peer  Reviewed  - Technote 

(Refer  to  Footnote  2 on  page  3 for  definitions  of  the  product  types.) 

External 

Barriers 

Advocacy 

A measure  that  indicates  the  involvement  of  advocacy  and  lobby  groups. 

Election 

A measure  that  indicates  whether  the  release  of  the  HTA  product 
coincided  with  the  provincial  election  held  in  March  2001 . 

Other  External 
Barriers 

A measure  that  indicates  whether  other  external  barriers  may  have 
negatively  influenced  the  impact  of  the  HTA  product. 

Internal 

Enablers 

Supporting 

Organizational 

Structures 

A measure  that  indicates  whether  the  requesting  organization  or  individual 
had  established  decision  making  structures  (e.g.,  standing  or  ad  hoc 
committees)  and/or  communication  channels  and  networks  in  place. 

Process 

Enablers 

Procedures 

A measure  that  indicates  whether  specific  aspects  of  the  process  for 
requesting  and  developing  the  product  positively  influenced  its  impact. 

Process 

Barriers 

Timing 

A measure  that  indicates  that  the  timing  of  the  release  of  the  product 
negatively  influenced  its  impact 

Immediate 
Outcome  and 
Future 
Potential 

Impact 

A measure  that  indicates  the  degree  to  which  the  HTA  product  informed 
decision-making  or  will  inform  future  decisions. 
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Discussion 


All  organizations  that  commissioned  HTA  products  required  evidence  to  inform 
decisions.  Alberta  Health  and  Wellness  was  involved  in  five  of  the  ten  products 
reviewed  because  they  needed  evidence  to  make  decisions  on  funding  for  individual 
claims  and  in  some  cases  whether  to  publicly  fund  the  treatment  in  question.  Alberta 
Children's  Services  requested  two  products  to  assist  them  in  making  decisions  on 
program  funding  and  evaluating  the  elements  of  programs  and  Capital  Health 
Authority  requested  a product  for  similar  reasons.  Alberta  Medical  Association 
requested  a product  to  inform  decisions  about  changes  to  clinical  practice  guidelines. 
Finally,  a product  was  requested  to  inform  decisions  about  care  for  an  individual 
patient.  As  discussed  below,  the  degree  to  which  the  products  influenced  decision- 
making varied. 

Product  Attributes 

Characteristics  of  the  products  themselves  influenced  the  impact  they  had  on  decision- 
making. Products  that  presented  evidence  with  clear  conclusions  tended  to  have  a 
greater  influence  on  decision  making  than  those  that  did  not.  Two  of  the  products 
presented  conclusive  evidence  but  did  not  have  any  impact  on  decisions.  This  is 
because  for  these  two  products,  the  product  was  not  available  until  after  the  decision 
had  been  made.  There  does  not  appear  to  be  any  relationship  between  whether  a 
product  was  peer  reviewed  (i.e.,  the  type  of  report)  and  its  impact. 

External  Barriers/Enablers 

Circumstances  external  to  requesting  organizations  may  also  influence  the  ability  of 
organizations  to  use  HTA  products  in  decision-making.  In  three  instances,  interviewees 
suggested  that  barriers  external  to  their  organizations  limited  their  ability  to  realize  the 
full  impact  of  the  conclusions  in  the  product.  The  most  frequently  reported  barrier  was 
lobbying  by  advocacy  groups.  Other  external  barriers  reported  were:  precedent  set  by  a 
court  ruling;  practices  of  the  current  program  and  of  other  provinces;  and,  difficulty  in 
evaluating  programs  that  do  not  match  those  in  the  research  literature.  None  of  the 
interviewees  reported  external  enablers. 

Lehoux  et  al.7  describe  the  relationship  of  an  HTA  product  to  its  environment  as  one  of 
"turbulence"  or  "continuity".  Turbulence  occurs  when  the  conclusions  of  the  product 
conflict  with  the  dominant  view  of  stakeholders  and  continuity  occurs  when  the 
conclusions  of  the  product  are  congruent  with  the  views  of  the  stakeholders.  The 
Lehoux  study  suggests  that  the  influence  of  an  HTA  product  is  likely  to  be  greater 
under  conditions  of  continuity  rather  than  turbulence.  Four  of  the  products  examined 
in  this  study  were  released  into  environments  that  included  active  lobbyist  advocacy; 
but  only  two  of  these  instances  could  be  considered  turbulent  situations  in  that  the 

Lehoux,  P.,  R.N.  Battista  and  J.-M.  Lance  (2000).  Monitoring  Health  Technology  Assessment  Agencies.  The 
Canadian  Journal  of  Program  Evaluation:  15(2)  1-33. 
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evidence  presented  in  these  two  products  opposed  the  position  of  the  advocates. 
However,  the  cases  of  these  two  products  challenge  the  Lehoux  hypothesis  since  the 
decision  made  opposed  the  view  of  the  advocacy  group  and  reflected  the  evidence 
presented  in  the  product.  In  the  cases  of  the  other  two  products,  the  evidence  presented 
by  the  products  was  inconclusive  (i.e.,  neither  supported  or  opposed  the  dominant 
view)  and  therefore  cannot  be  considered  as  either  turbulent  or  continuous. 

Interviewees  associated  with  these  latter  two  products  suggested  that  the  presence  of 
advocacy  activity  in  combination  with  inconclusive  evidence  lessened  the  impact  of  the 
products. 

Internal  Barriers/Enablers 

Very  few  of  the  interviewees  reported  internal  barriers  that  may  have  adversely 
affected  their  ability  to  use  evidence  in  the  HTA  product  to  make  decisions.  Rather, 
almost  all  had  enabling  structures  in  place  to  facilitate  decision-making.  Most 
organizations  that  requested  products  either  had  standing  or  ad  hoc  committees  that 
had  been  charged  with  the  decision-making  responsibility  and  many  had  established 
communication  networks  to  inform  stakeholders  of  the  evidence  presented  in  products. 

Process  Barriers/Enablers 

The  major  process  barrier  to  impact  of  HTA  products  was  timing  of  the  delivery  of  the 
products.  In  two  cases  decisions  were  made  before  the  products  became  available  and, 
as  a result,  the  products  had  no  impact,  at  least  as  reported  by  the  interviewees  in  this 
study.  In  another  two  instances,  the  requesting  organizations  postponed  their  decisions 
until  the  HTA  products  became  available.  Interviewees  recognized  that  the  HTA 
products  take  a great  deal  of  time  to  prepare  especially  when  evidence  must  be 
retrieved  from  specialist  and  foreign  language  journals.  Also,  many  contextual  factors 
influence  timing  and  may  affect  requester  needs  over  the  course  of  the  HTA  production 
process. 

All  requesters  of  the  products  felt  that  the  HTA  process  and  product  facilitated  their 
ability  to  make  decisions.  Interviewees  reported  that  the  products  of  the  HTA  Unit  are: 
of  high  quality,  thoroughly  researched  and  perceived  by  those  who  use  them  as  being 
unbiased  and  objective.  The  reputation  and  credibility  of  the  HTA  Unit  was  cited  as 
having  a positive  impression  and  impact  on  expert  committees.  Availability  of  reports 
at  the  time  of  decision  was  also  seen  as  an  important  factor  in  enabling  impact. 

All  requesters  of  the  HTA  products  thought  that  the  HTA  Unit  provided  a valuable 
service  and  several  suggested  ways  that  the  HTA  Unit  could  expand  their  services  to 
better  meet  the  needs  of  their  clients.  Several  requesters  mentioned  that  they  would 
request  an  updated  product  in  the  future  and  one  suggested  that  the  HTA  Unit  should 
consider  regularly  updating  products  on  high  priority  topics.  Two  organizations  would 
like  to  see  the  HTA  Unit  take  a more  active  role  in  disseminating  their  products  and  two 
would  like  the  HTA  Unit  to  provide  short  products  with  a quick  turnaround  time;  i.e., 
three  to  seven  days. 
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It  should  be  noted  that  the  HTA  Unit  does  provide  some  of  these  products  and  services 
but  it  appears  that  its  clients  are  not  always  aware  of  the  Unit's  range  of  products.  For 
example,  the  HTA  Unit  has  a policy  that  articulates  to  whom  they  disseminate 
information  in  their  products8;  and  with  respect  to  turnaround  time,  the  HTA  Unit 
produces  a product  called  an  "Information  Letter"9  that  is  designed  to  meet  this  need. 
One  interviewee  suggested  that  the  HTA  Unit  may  like  to  consider  developing  a 
"menu"  of  products  and  services  that  outlines  the  options  available  for  each  product. 

Impact 

Of  the  ten  products  considered  in  this  study,  eight  informed  policy  and  resource 
allocation  decisions  to  some  extent.  Besides  decision  making,  the  original  requesters  of 
the  products  also  used  them  to  raise  awareness  about  the  health  technology  itself  or 
issues  in  dealing  with  patients  who  present  specific  conditions;  provide  impetus  and 
ideas  for  future  research;  and,  change  elements  of  a program  (Table  4:  Immediate 
Outcomes).  Many  of  the  original  requesters  shared  the  information  in  the  products  or 
the  products  themselves  with  colleagues  or  other  individuals  and  organizations  who 
requested  information  about  some  aspect  of  the  health  technology.  These  secondary 
uses  included  providing  evidence  for  use  in  legal  decisions;  educating  practitioners  and 
health  system  administrators;  and,  providing  information  to  other  health  jurisdictions. 
Because  of  the  secondary  uses  of  the  products,  it  is  difficult  to  measure  the  full  impact 
of  the  HTA  products  (see  Limitations  below.) 


s The  distribution  channels  for  the  product  depend  upon  the  type  of  product  (i.e.,  whether  is  an  HTA  Report,  HTA 
Brief  or  Technote). 

’ An  Information  Letter  is  a two-three  page  letter  based  on  a review  of  abstracts  only,  sometimes  including  a 
reference  list  of  potentially  useful  articles  and  web  sites.  Turnaround  time  for  this  product  is  typically  one  week. 
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Limitations 

In  this  study  AHFMR  was  exploring  the  possibility  of  assessing  the  impact  of  the 
products  of  the  HTA  Unit.  As  such,  its  scope  was  limited  to  interviewing  individuals 
who  requested  products  from  the  HTA  Unit.  Due  to  the  scope  of  the  study,  there  are 
inherent  limitations  in  its  findings  and  conclusions. 

Underestimates  the  impact  of  the  HTA  products 

The  only  people  interviewed  in  this  study  were  those  who  directly  requested  the 
2000-2001  HTA  products.  However,  the  products  have  all  been  widely  disseminated 
and  the  study  did  not  attempt  to  identify  (and  interview)  all  organizations  and 
individuals  who  may  have  used  them  to  inform  decision-making.  Furthermore,  all  of 
the  products  considered  in  this  study  had  been  published  in  the  previous  fiscal  year  to 
the  study  (i.e.,  they  were  relatively  "new"  products).  Consequently,  they  may  have 
other  anticipated  and  unanticipated  impacts  in  the  future.  Clearly,  the  products  have 
value  in  providing  evidence  to  inform  decisions  beyond  that  which  they  were  originally 
intended  (i.e.,  to  meet  the  needs  of  the  original  requesters).  The  conclusions  of  this 
study  likelv  underestimate  the  impact  of  HTA  products. 

Does  not  specifically  assess  changes  in  practitioner  behaviour  and 
system  outcome 

Ultimately  (and  theoretically),  the  decisions  informed  by  the  HTA  products  should 
result  in  changes  in  the  health  care  system,  practitioner  behaviour,  and  patient 
outcomes.  However,  this  study  did  not  consider  the  impact  of  the  HTA  products  at 
these  levels.  It  was  limited  in  its  ability  to  do  so  for  three  reasons:  (1)  most  of  the  people 
interviewed  were  administrators  charged  with  making  funding  and  policy  decisions;  (2) 
all  the  products  had  been  commissioned  within  the  past  18  months  which  is  likely 
insufficient  time  for  these  outcomes  to  be  realized;  and,  (3)  these  outcomes  are  very 
difficult  to  measure. 

However,  the  interview  list  included  two  practitioners,  both  of  whom  were  involved  in 
the  same  HTA  product  request.  These  individuals  indicated  the  product  contained 
useful  information  about  barriers  to  treating  patients  in  their  field  of  specialization. 

Each  uses  the  information  in  the  product  in  their  own  practice  and  research  program. 
Even  though  the  study  did  not  specifically  identify  changes  to  practitionerbehaviour 
and  patient  outcome,  this  little  evidence  suggests  that  the  HTA  products  are  having  an 
impact,  even  after  being  available  for  less  than  18  months. 
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Does  not  attempt  to  identify  relative  contributions  of  the  HTA  product 
in  making  decisions. 

Some  interviewees  indicated  that  the  decision-makers  used  information  and  evidence 
other  than  that  provided  in  the  HTA  products  in  making  decisions.  Since  many  of  the 
interviewees  were  not  the  decision-maker,  they  could  not  estimate  the  relative 
importance  of  the  HTA  products  in  making  the  decisions.  Even  for  those  who  were 
decision-makers,  it  was  difficult  for  them  to  reconstruct  which  sources  of  evidence 
influenced  their  decisions. 
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Conclusions 


• Products  that  present  inconclusive  evidence  have  less  impact  on  decision-making 
than  those  that  present  conclusive  evidence.  (It  should  be  noted,  however,  that  the 
nature  of  the  evidence  that  is  used  to  develop  a particular  HTA  product  is  outside 
the  control  of  the  HTA  Unit.) 

• The  presence  of  external  barriers  (i e.g .,  advocacy  actions  of  lobby  groups  and 
momentum  of  existing  programs)  can  limit  the  impact  of  products. 

• Timing  of  HTA  product  availability  and  timing  of  the  client  request  and  decision  is 
important  in  influencing  impact,  as  impact  is  increased  if  the  product  is  available  at 
the  time  the  related  decision  is  being  made.  Timing  itself  appears  to  be  influenced 
by  the  client,  the  HTA  Unit,  and  various  other  factors  (e.g.,  external  factors  may 
require  HTA  clients  to  make  decisions  sooner  than  was  initially  anticipated). 

• Characteristics  of  the  product  development  process  that  appeared  to  enhance  their 
impact  on  decision-making  are: 

> High  quality,  thorough  and  unbiased  research; 

> Products  are  written  at  a level  appropriate  for  the  audience;  and, 

> Reputation  and  credibility  of  HTA  Unit. 

• HTA  Unit  clients  are  not  fully  aware  of  the  range  of  products  and  services  offered  by 
the  HTA  Unit  or  the  options  available  for  the  various  products. 

• Evidence  presented  in  HTA  products  informs  decisions  of  the  requesting 
organization.  The  scope  of  influence  is  difficult  to  assess  because  the  products  have 
been  widely  distributed  and  the  conclusions  in  them  are  potentially  being  used  to 
inform  decisions  beyond  which  they  were  originally  intended. 
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Appendix  1 : Questions  Used  to  Guide  the  Interview 

Note  that  your  name  will  not  be  used  in  the  report  submitted  to  AHFMR.  The  report 
will  summarize  findings  for  specific  HTA  reports  and  will  draw  general  conclusions 
about  variables  that  could  influence  the  impact  of  a report;  e.g.,  "we  found  that  health 
regions  were  more  able  to  implement  the  conclusions  of  reports  where  they  had 
adequate  resources  to  do  so". 

Introduction 

1.  What  was  vour  role  in  relation  to  this  report? 

Context 

2.  Why  was  the  HTA  report  requested  and  for  whom  was  it  intended? 

3.  When  requesting  the  report,  what  did  you  expect  the  impact  of  the  report  to  be? 

4.  Were  these  expectations  met?  If  not  why? 

5.  Were  there  internal  barriers  within  your  organization  to  implementing  the 
conclusions  contained  within  the  report?  Were  there  any  external  barriers? 

6.  What  strategies  did  you  put  in  place  to  prepare  the  stakeholders  in  your 
organization  for  acceptance  of  the  conclusions  of  the  report? 

Process 

7.  Were  there  any  specific  aspects  of  the  process  for  requesting  or  developing  the 
HTA  report  that  you  feel  influenced  (or  will  influence)  the  impact  of  the  report? 
Consider  both  positive  and  negative  influence  on  the  impact. 

Outcomes 

8.  How  are  you  using  the  report?  What  did  you  do  with  it  when  you  received  it? 
How  will  the  conclusions  be  used? 

9.  Does  the  report  have  future  potential  for  your  organization?  If  so,  what  changes 
do  you  expect  to  see  and  in  what  time  frame  do  you  expect  the  potential  to  be 
realized? 

Examples  of  changes: 

o awareness  of  the  technology 
o policy 

o resource  allocation  and  priorities 
o patient  outcomes 
o future  research 
o other 

10.  Is  there  anything  else  we  should  have  asked,  but  didn't? 
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Appendix  2:  Sample  Email  Sent  to  Prospective  Interviewees 
by  HTA  Unit 


Hi  <first  name> 

I am  contacting  you  to  invite  you  to  participate  in  one  of  the  Health  Technology 
Assessment  Unit's  impact  analysis  and  continuous  improvement  projects.  You  have 
been  chosen  to  participate,  as  you  were  one  of  our  main  clients  for  the  project  on  <name 
of  project>  during  the  fiscal  year  2000/2001. 

As  you  may  know,  the  Unit's  Report  of  Activities  for  1999-2000  (see 
http://www.ahfmr.ab.ca/hta/hta-publications/annual/99-00-web-report.pdf) 
attempted  to  identify  the  impact  of  the  unit's  products,  produced  during  the  course  of 
that  fiscal  year,  on  the  practice,  decisions  and/ or  policy  of  the  clients  who  had 
requested  the  work.  The  impacts  that  were  identified  for  that  report  were  descriptive 
and  brief. 

The  objective  of  this  current  project  is  to  develop  a further  sophistication  in  our 
understanding  and  exposition  of  the  impact  of  the  HTA  Unit's  products.  To  that  end, 
we  have  engaged  the  services  of  a consultant.  Turnkey  Management  Consulting.  Data 
collection  will  consist  of  interviews  with  our  key  client  contacts.  The  results  of  the 
project  will  guide  the  continuous  improvement  efforts  of  the  Unit,  will  be  included  in 
the  2000-2001  Report  of  Activities,  and  may  be  shared  with  other  HTA  agencies  and 
HTA  Unit  stakeholders.  This  approach  will  also  assure  a more  unbiased  process  and 
perhaps  allow  a more  open  dialogue.  It  is  also  expected  that  this  work  will  serve  as  a 
template  to  be  repeated  in  following  years. 

Before  November  30,  2001  a Turnkey  consultant  will  be  contacting  you  to  schedule  a 
brief  telephone  or  in-person  interview.  Your  responses  will  be  held  in  the  strictest 
confidence  and  no  interviewee  names  will  be  included  in  the  Final  Report.  You  are,  of 
course,  under  no  obligation  to  participate  in  this  project  but  we  would  greatly 
appreciate  your  input. 

If  you  have  any  concerns  about  any  aspect  of  this  project,  please  contact  Richard 
Thornley,  AHFMR's  Coordinator,  Impact  Analysis  at  (780)  423-5727,  or 
r ichar  d . thor  nley  @ahf mr . ab . ca . 

Best  regards. 


Christa 
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